We use four layered clip encoder in our experiment. See Table 2. Pretraining on large amount of data yields better top-1 We further compare the ViL T with convolution variants and one Transformer variant, i.e., LSH attention. We compare ViL T (78.4%, 98.3%) with various convolution We have convolution (73.9%, 94.9%), Empirically, we compare the shifted MSA with various attentions, i.e., space attention (conventional From the perspective of human vision system, the typical duration of persistence of vision is 0.1-0.4s. The shifted MSA is forced to learn fine-grained motion information.

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences. The Transformer architecture (V aswani et al., 2017) is widely used in natural language processing and yields state-of-the-art results on a number of tasks. To obtain these results, researchers have resorted to training ever larger Transformer models. The number of parameters exceeds 0.5B per layer in the largest configuration reported in (Shazeer et al., 2018) while the number of layers goes up to 64 in (Al-Rfou et al., 2018). Transformer models are also used on increasingly long sequences.